Papers with Spider benchmark
Improving Generalization in Semantic Parsing by Increasing Natural Language Variation (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches generate question reformulations via models trained on Spider or only introduce local changes. Existing methods generate question form reformulation but lack robustness. |
| Approach: | They use data augmentation to enhance the robustness of text-to-SQL parsers against natural language variations by generating more realistic and diverse questions. |
| Outcome: | The proposed model improves on the new spider dataset by using a few prompts. |
Evaluating Cross-Domain Text-to-SQL Models and Benchmarks (2023.emnlp-main)
Copied to clipboard
| Challenge: | Text-to-SQL benchmarks are used to evaluate progress made in the field . however, matching a model-generated SQL query to a reference SQL query fails due to various reasons. |
| Approach: | They conduct an extensive evaluation of text-to-SQL benchmarks and re-evaluate some of the top-performing models. |
| Outcome: | The results show that a recent model surpasses the gold standard reference queries in the Spider benchmark in human evaluation. |
Declarative Techniques for NL Queries over Heterogeneous Data (2025.emnlp-industry)
Copied to clipboard
Elham Khabiri, Jeffrey O. Kephart, Fenno F. Heath Iii, Srideepika Jayaraman, Yingjie Li, Fateh A. Tipu, Dhruv Shah, Achille Fokoue, Anu Bhamidipaty
| Challenge: | In many industrial settings, users wish to ask questions in natural language . however, these applications do not cope with data source heterogeneity that typifies such environments. |
| Approach: | They propose a declarative approach to handling data heterogeneity in industrial settings . they simulate the heterogenity of industrial environments by adding two extensions of the popular Spider benchmark dataset . |
| Outcome: | The proposed approach copes with data source heterogeneity better than state-of-the-art systems. |
Towards Robustness of Text-to-SQL Models against Synonym Substitution (2021.acl-long)
Copied to clipboard
Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, Pengsheng Huang
| Challenge: | Existing text-to-SQL models rely on lexical matching between words in NL questions and tokens in table schemas, which may break the schema linking mechanism. |
| Approach: | They propose a human-curated dataset for text-to-SQL translation . they replace schema-related words with manually selected synonyms . |
| Outcome: | The proposed model outperforms its counterparts without the defense. |
Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing text-to-SQL models do not generalize when faced with domain knowledge that does not frequently appear in training data. |
| Approach: | They propose a human-curated dataset based on the Spider benchmark for text-to-SQL translation. |
| Outcome: | The proposed model performs better on unseen domains than existing models on public benchmarks. |
DCG-SQL: Enhancing In-Context Learning for Text-to-SQL with Deep Contextual Schema Link Graph (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for Text-to-SQL show little improvement compared to random selections . Existing approaches rely on intrinsic capabilities of hyper-scaled LLMs, not useful demonstrations. |
| Approach: | They propose a novel approach to effectively retrieving demonstrations and generating SQL queries by linking a question and its database schema items. |
| Outcome: | The proposed method shows consistent improvements in performance and efficiency across hyper-scaled LLMs and small LLM. |